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Abstract 


In  early  2002,  the  Communications  Electronics  Command  (CECOM)  Manager  of  the  Army 
Tactical  Wireless  Network  Assurance  (TWNA)  Science  and  Technology  Objective  (STO) 
FY03-07,  hereafter  referred  to  as  STO,  requested  assistance  from  the  Software  Engineering 
Institute  (SEI)  in  improving  STO  methods  for  assessing  the  maturity  of  new  information- 
assurance  technologies.  The  STO  was  seeking  to  use  technology  maturity,  as  measured  by  the 
Technology  Readiness  Levels  (TRLs)  scale,  as  a  metric  in  its  decision-making  process  for 
selecting  new  technologies  for  STO  development  and  maturation,  technologies  that  would 
eventually  be  transitioned  to  Army  tactical  programs.  This  report  describes  the  results  of  the 
SEI  study  of  the  feasibility  of  (a)  using  TRLs  in  STO  technology  screening,  (b)  developing  or 
acquiring  a  TRL  tool,  and  (c)  implementing  a  TRL  tool. 
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IX 


1  Project  Overview 


1 .1  Objectives 

The  objective  of  the  project  documented  in  this  report  was  to  assess  the  feasibility  of  devel¬ 
oping  an  information  assurance  (IA)  technology  readiness  level  (TRL)  assessment  method  (or 
equivalent)  for  technologies  at  various  TRLs.  The  purpose  of  such  an  assessment  method  is 
to  assist  the  Communications  Electronics  Command  (CECOM)  Manager  of  the  Army  Tacti¬ 
cal  Wireless  Network  Assurance  (TWNA)  Science  and  Technology  Objective  (STO),  and 
others  in  the  S&T  community,  in  identifying  technologies  in  basic  research  and  applied  re¬ 
search  categories  that  would  benefit  the  Army  and  other  services.1  Effective  use  of  TRLs  can 
reduce  the  risk  associated  with  investing  in  immature  technologies. 


1 .2  Background 

In  early  2002,  the  CECOM  Manager  of  the  Army  TWNA  STO  FY03-07,  hereafter  referred  to 
as  STO,  requested  the  Software  Engineering  Institute’s  (SEI’s)  assistance  in  improving  their 
methods  for  assessing  and  identifying  the  maturity  of  new  IA  technologies.  The  premise  was 
that  technology  maturity  is  a  useful  metric  in  the  decision-making  process  of  selecting  new 
technologies  for  development  and  maturation  within  an  advanced  technology  demonstration 
(ATD)  or  STO  environment,  technologies  that  will  eventually  be  transitioned  to  Army  tactical 
programs.  The  SEI  and  the  Manager  of  the  STO  developed  a  plan  for  three  phases  of  work, 
the  first  of  which,  a  feasibility  study,  began  in  February  2002  and  is  the  subject  of  this  report. 

The  overall  goal  of  the  project  (the  cumulative  result  of  all  three  phases)  is  to  provide  the 
STO  with  an  IA  TRL  assessment  method,  or  an  equivalent,  to  support  their  decision-making 
process  in  selecting  new  technologies  for  investment.  Recent  DoD  regulations  (see  Appendix 
A:  Technology  Readiness  Levels)  have  put  an  emphasis  on  the  use  of  TRLs  to  improve  the 
ability  of  acquisition  programs  to  select  mature  technologies  for  inclusion  in  their  programs. 
The  STO  is  seeking  to  apply  this  same  or,  at  a  minimum,  a  compatible  approach  to  aid  their 
investment  decisions,  which  occur  early  in  a  technology’s  life  cycle.  The  difference  between 
the  two  domains  is  that  the  STO  (or,  likewise,  an  ATD)  generally  seeks  to  invest  in  technolo¬ 
gies  at  TRL  4  or  so,  while  acquisition  program  managers  generally  seek  technologies  at  TRL 
6  or  higher.  One  of  the  goals  of  the  STO  (and  ATD)  is  to  mature  technologies  that  are  TRL  4 

1  Calendar  Year  2002  Work  Plan  for  the  Tactical  Information  Assurance  STO  FY03-07,  PWS  4-357, 
Version  2.1,  Software  Engineering  Institute. 
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or  below  to  at  least  a  TRL  6,  making  the  technologies  more  mature  and  thus  more  “ready”  for 
insertion  into  acquisition  programs. 


STO  is  expected  to  mature  new  information  assurance  technologies  to  DoD  TRL  6  in  no 
more  than  four  to  five  years  after  project  start.  The  lower  the  maturity,  or  readiness,  of  an  in¬ 
coming  technology  (the  technology  selected  by  STO  for  maturation),  the  more  time  and 
money  will  likely  be  needed  to  mature  that  technology  to  TRL  6.  Thus,  STO  considers  a  TRL 
4  or  5  to  be  the  minimum  acceptable  readiness  level  for  an  incoming  technology  that  will  en¬ 
able  them  to  satisfy  their  program  constraints.  STO  is  looking  for  consistent,  repeatable 
methods  to  evaluate  technology  readiness  as  part  of  their  investment  decision-making  proc¬ 
ess,  to  self-assess  their  own  technology,  and  to  help  filter  all  the  possible  technologies  down 
to  the  most  promising  ones.  CECOM  has  an  ongoing  task  to  weed  through  the  many  pro¬ 
grams  to  find  ones  relevant  to  their  mission.  Once  identified  as  relevant  to  the  mission,  a  TRL 
estimate  can  help  reduce  the  risk  of  investing  in  a  technology  that  is  “too  immature.” 

While  the  DoD  has  issued  new  regulations  stating  that  new  major  programs  must  utilize 
TRLs  or  a  yet-to-be-identified  equivalent,  there  has  apparently  been  no  top-level  guidance  on 
how  to  determine  a  TRL  (our  interviews  with  several  TRL  users  from  the  General  Accounting 
Office  (GAO),  Air  Force  Research  Lab  (AFRL),  and  Defense  Advanced  Research  Projects 
Agency  (DARPA)  indicated  this  indeed  to  be  the  case).  This  project  is  an  attempt  to  provide 
some  of  this  missing  guidance,  specifically  for  support  of  STO  needs,  by  understanding  what 
methods,  tools,  or  techniques  others  are  currently  using  to  estimate  the  TRL  of  a  given  tech¬ 
nology. 


1 .3  Approach 

This  feasibility  study  sought  to  answer  this  question: 

Is  it  feasible  to  develop  (or  acquire  if  available)  a  TRL  (or  equivalent)  tool  (such  as  a 
checklist  or  software  package )  that  enables  the  Army  STO  Manager  to  assess  the  ma¬ 
turity  of  new  I  A  technologies? 


The  detailed  project  plan  is  provided  in  Appendix  C.  In  general,  our  approach  included  iden¬ 
tifying  the  needs  of  the  STO  regarding  TRL  use,  assessing  the  state  of  the  practice  of  other 
TRL  users,  and  synthesizing  the  results  into  the  following  categories  of  findings,  which  are 
detailed  in  the  Findings  section  of  this  report: 

•  Conceptual  Feasibility:  Is  it  feasible  for  TRLs  (or  an  equivalent)  to  support  or  improve 
the  STO  Manager’s  decision-making  process  in  selecting  new  LA  technologies  for  ATD 
investment? 

•  Development  Feasibility:  What  resources  would  be  needed  for  the  development  of  such  a 
tool  if  it  does  not  already  exist,  or  for  its  adaptation  for  use  by  the  STO  if  such  a  tool  does 
exist? 
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TRL  Tool  Implementation  Feasibility:  What  resources  are  needed  to  employ  such  a  tool 
in  the  Army  STO  and  DARPA  S&T  communities? 


1.4  Scope  and  Constraints 

A  feasibility  study  can  range  in  scope  from  a  few  months  to  more  than  a  year,  depending  on 
the  complexity  of  the  issues  being  studied.  To  limit  the  duration  and  budget  of  this  study,  the 
following  project  constraints  were  agreed  to  at  the  project  kickoff  meeting  in  February  2002: 

•  Concentrate  on  assessing  the  technology’s  readiness  coming  into  the  STO  (i.e.,  is  it  ready 
for  the  STO  to  take?),  rather  than  the  readiness  of  the  technology  going  out  of  the  STO 
(i.e.,  is  it  ready  for  the  STO  to  transition  to  product  developers?).  However,  the  outbound 
technology  readiness  is  a  closely  related  issue  and  much  of  what  is  discussed  herein  ap¬ 
plies  in  that  case  as  well. 

•  If  the  feasibility  study  identifies  potential  alternatives  to  the  TRLs,  they  can  be  reported. 
However,  because  of  the  Army’s  interest  and  emphasis  on  TRLs  as  they  are  currently  de¬ 
fined,  it  would  require  some  “sales  and  negotiation”  to  convince  others  that  there  is  a  bet¬ 
ter  way.  If  an  alternative  looks  more  suitable,  the  practical  approach  would  be  to  map  to 
or  package  it  in  TRL  terms. 

1.5  Challenges 

One  of  the  main  challenges  of  providing  a  readiness  assessment  method  to  support  the  STO  is 
tied  to  their  relationship  with  DARPA.  DARPA  is  a  major  source  of  new  IA  technologies  for 
STO,  with  a  significant  number  of  IA  projects  currently  underway.  While  the  STO  personnel 
that  we  interviewed2  stated  that  they  have  a  good  rapport  with  DARPA  Program  Managers 
(PMs)  and  some  of  the  Principal  Investigators  (Pis),  it  would  be  challenging  indeed  to  meet 
with  roughly  50  DARPA  Pis  (the  typical  number  of  projects  annually  evaluated  by  STO)  do¬ 
ing  research  for  DARPA  on  a  regular  basis  to  discuss  the  readiness  of  each  technology.  Two 
STO  interviewees2  verified  this,  stating  that  it  is  difficult  to  get  on  the  DARPA  Pis’  calendars. 
To  facilitate  awareness  and  understanding  of  their  technologies,  DARPA  holds  conferences 
throughout  the  year.  According  to  the  STO  interviewees,  about  15  technologies  are  described 
in  presentations,  about  15  are  demonstrated,  and  the  rest  are  reflected  in  document  form. 

Even  in  this  venue,  however,  the  STO  interviewees  said  they  find  it  labor-intensive  to  assess 
technology  readiness  because  that  metric  is  not  consistently  expressed  in  research  and  there¬ 
fore  must  be  somehow  extracted  or  formulated  from  the  information  Pis  provide.  The 
DARPA  PMs  we  interviewed  for  this  report,3  however,  stated  that  the  information  presented 
at  these  meetings  generally  doesn’t  contain  sufficient  detail  for  estimating  TRLs. 


2  Conversations  with  Robert  J.  Schenk,  U.S.  Army  CECOM  RDEC  Space  and  Terrestrial  Commu¬ 
nications  Directorate,  TWNA  STO  and  Peter  Van  Syckle,  U.S.  Army  CECOM  RDEC  Space  and 
Terrestrial  Communications  Directorate,  TWNA  STO. 

3  Conversations  with  Doug  Maughan,  DARPA  and  Jay  Lala,  DARPA. 
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In  summary,  the  challenges  to  providing  a  readiness  assessment  method  to  support  the  STO 
technology  selection  process  include  the  following: 

•  Regular  face-to-face  meetings  between  STO  personnel  and  DARPA  Pis  are  limited,  creat¬ 
ing  reliance  on  the  conference  materials  for  determining  a  TRL. 

•  The  technology  information  that  DARPA  provides  comes  in  several  forms  (presentations, 
demonstrations,  papers)  but  generally  does  not  contain  sufficient  information  to  allow 
STO  to  determine  TRLs  just  from  the  materials. 

•  Consistent  DoD  guidance  on  how  to  assess  TRLs  is  lacking,  thus  putting  STO  in  the  po¬ 
sition  of  defining  a  consistent  method  for  this  type  of  assessment. 

As  a  result,  STO  will  be  breaking  new  ground  in  including  TRL  information  in  their  decision¬ 
making  process  that  may  hopefully  help  others  in  the  S&T  community. 
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2  Findings 


2.1  A  Note  About  TRLs 

TRLs  are  described  in  the  DoD  5000.2-R  document  (see  Appendix  A)  from  a  systems  per¬ 
spective,  and  thus  are  intended  to  be  appropriate  for  both  hardware  and  software.  The  docu¬ 
ment  also  states  “DoD  Components  may  provide  additional  clarifications  for  software.”  The 
Army,  for  example,  has  developed  a  mapping  of  the  TRLs  to  software  (see  Appendix  B),  and 
the  Army  Medical  Research  and  Materiel  Command  is  working  on  defining  corollaries  for 
biomedical  TRLs.4  Thus,  TRLs  are  meant  to  be  overarching  definitions  for  any  technology, 
while  interpretations  or  amplifications  for  specific  technologies  are  left  to  the  experts  in  that 
technology  domain. 


2.2  Conceptual  Feasibility 

This  section  of  the  report  provides  our  findings  in  response  to  this  question  (see  Section  1.3): 
Is  it  feasible  for  TRLs  (or  an  equivalent)  to  support  or  improve  the  STO  Manager’s 
decision-making  process  in  selecting  new  I  A  technologies  for  investment? 

Currently,  TRLs  are  being  used  successfully  at  the  completion  of  an  ATD  or  STO  when  get¬ 
ting  ready  to  transition,  i.e.,  when  briefing  progress  to  the  Army  sponsor,  rather  than  as  a 
screening  approach  for  selecting  new  technologies.  Most  of  the  literature  on  TRLs  that  the 
SE1  team  surveyed  is  limited  to  the  context  of  using  TRLs  to  improve  the  timing  of  transi¬ 
tioning  or  inserting  a  technology  from  an  ATD-Iike  environment  to  a  product  development 
program  (acquisition  program).  We  found  no  literature  describing  the  use  of  TRLs  at  the  front 
end  of  an  ATD,  i.e.,  in  the  ATD  technology-selection  process.  We  note  that  the  results  of  this 
project  may  also  help  refine  the  readiness  assessment  process  at  the  TRL  6  stage  as  well.  The 
GAO  states  that  a  major  purpose  of  TRLs  is  to  “reveal  the  gap  between  a  technology’s  matur¬ 
ity  and  the  maturity  demanded  for  successful  inclusion  in  the  intended  product” 
[GAO/NSIAD  99].  TRL-related  requirements  are  integrated  into  the  ATD/STO  exit  require¬ 
ments.  Meeting  these  requirements  automatically  satisfies  a  ‘TRL  6.”  TRLs  are  not  the  only 
exit  criteria,  of  course.  Domain-specific  requirements,  such  as  preventing  or  detecting  net¬ 
work  security  events  a  certain  percentage  of  the  time,  are  obviously  critical. 


4  “Biomedical  Technology  Readiness  Levels  (TRLs),”  a  working  paper  provided  to  the  SEI  by  the 
U.S.  Army  Medical  Research  and  Materiel  Command,  but  not  approved  for  public  release. 
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While  the  literature  we  surveyed  and  the  TRL  users  we  talked  to  are  heavily  oriented  to  using 
TRLs  in  the  product  development  phase  (assessing  the  risks  of  including  a  technology  in 
product  development),  the  GAO’s  1999  report  [GAO/NS1AD  99]  cites  the  AFRL  as  having 
adapted  and  using  TRLs  “to  measure  the  key  steps  in  the  progression  of  technology  from  ini¬ 
tial  concept  to  proven  performance,”  thus  indicating  the  use  of  TRLs  throughout  a  technology 
development  cycle.  Conversations  with  William  Nolte  at  AFRL  confirm  this  statement.  AFRL 
is  currently  using  TRLs  on  a  DARPA  project  (Medusa),  which  is  being  managed  by  AFRL. 


At  the  front  end  of  the  STO  technology  management  life  cycle,  current  practice  for  selecting 
new  technologies  involves  a  variety  of  parameters,  though  generally  not  yet  TRLs.  Some  of 
the  parameters  factored  into  a  new  technology  selection  include 

•  the  applicability  of  the  technology  to  the  ATD  and  STO  program.  Obviously,  only  tech¬ 
nologies  that  satisfy  the  mission  of  the  program  are  suitable  candidates. 

•  availability  of  the  technology  developers.  Having  the  support  of  the  basic  and  applied 
research  and  technology  developers  available  to  the  STO  developers  throughout  the  STO 
project  can  be  critical,  and  in  some  cases  can  be  a  deal-breaker  if  the  STO  developers  are 
not  confident  that  such  support  will  be  present.  Access  to  the  primary  research  staff  is 
considered  a  critical  success  factor  in  preparing  the  technologies  for  product  development 
use.  As  a  result,  the  selection  process  generally  rejects  technologies  that  are  no  longer  be¬ 
ing  actively  developed.  Once  a  project  has  closed  down,  the  researchers  go  on  to  other 
work  and  are  not  available  to  support  their  original  findings. 

•  The  skills  of  the  technology  lead.  A  technology  lead  with  experience  in  the  technology 
domain,  good  human  interaction  skills,  and  a  sincere  interest  in  continuing  the  maturation 
of  his  or  her  technology  (via  the  ATD/STO  process)  is  an  important  factor  in  the  technol¬ 
ogy’s  ultimate  success.  Thus,  the  skills  of  the  technology  lead  are  taken  into  account  in 
the  ATD/STO  selection  process. 

•  Consistency  of  project  funding.  Technology  projects  with  consistent  and  sufficient  fund¬ 
ing  are  more  likely  to  be  objective  with  their  TRL  estimates  and  not  be  tempted  to  use 
them  as  a  sales  vehicle  to  secure  funding. 

So,  what  are  the  perceived  benefits  of  using  TRLs  in  STO  or  ATD  technology  selection?  ATD 
personnel  we  interviewed5  estimated  that  TRLs  may  address  approximately  30%  of  the  fac¬ 
tors  that  they  need  to  pay  attention  to  in  making  their  technology  selections.  They  serve  as  a 
risk-reduction  measure.  Domain-related  issues,  requirements  that  are  derived  by  CECOM, 
and  ATD  exit  criteria,  make  up  the  majority  of  the  selection  criteria.  If  this  estimate  is  accu¬ 
rate,  TRLs  can  be  said  to  bring  value  to  the  STO  or  ATD  technology-selection  process, 
though  they  should  be  considered  as  only  one  of  numerous  critical  decision  criteria.  In  addi¬ 
tion,  one  of  our  interviewees6  told  us  that  the  use  of  TRLs  across  the  research  and  develop¬ 
ment  community  encourages  the  ATD  to  build  from  existing,  though  immature,  technologies 


5  Conversations  with  Robert  Serenelli  and  Frank  Geek,  KeyWay  Security. 

6  Conversation  with  Peter  Van  Syckle,  U.S.  Army  CECOM  RDEC  Space  and  Terrestrial  Communi¬ 
cations  Directorate,  Tactical  Wireless  Network  Assurance  (TWNA)  Science  and  Technology  Ob¬ 
jective  (STO). 
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from  universities  or  research  labs  (e.g.,  Army  Research  Lab,  Naval  Research  Lab)  and  evolve 
them,  rather  than  developing  new  technologies  themselves  from  scratch.  Thus,  the  use  of 
TRLs  is  not  only  being  “strongly  encouraged”  by  senior  Army  and  DoD  officials,  it  also 
shows  merit  for  use  in  the  ATD/STO  technology-selection  process. 

What  about  equivalent  measures  of  technology  readiness  and  their  value  in  the  STO  technol¬ 
ogy  selection  process?  A  survey  of  more  than  30  articles  in  the  field  of  new  product  devel¬ 
opment  and  portfolio  management  in  industry  provided  little  in  the  way  of  equivalencies  at 
the  level  of  detail  at  which  TRLs  are  currently  defined.  The  closest  was  a  six-level  scale  of 
technology  maturity,  with  the  highest  level  of  maturity  denoting  commercialization  [TRECC 
01].  That  scale’s  purpose  is  to  identify  commercial  technologies  for  potential  adoption  by  the 
DoD.  The  literature  also  highlighted  the  fact  that  technology  maturity  (which  TRLs  are  in¬ 
tended  to  help  measure)  is  only  one  of  numerous  critical  factors  used  by  industry  to  select 
and  prioritize  technology  projects.  Our  interviews  with  ATD  personnel  (detailed  above)  con¬ 
firmed  a  similar  perception  of  the  value  of  TRLs  in  the  overall  technology  selection  process. 
One  article  [Heslop  01]  listed  more  than  50  factors  contributing  to  the  successful  transition  of 
technologies  from  research  universities  and  other  R&D  sources  into  commercial  use. 


Another  issue  to  consider  is  that  TRLs  are  currently  defined  for  system  technologies  (see  Ap¬ 
pendices  A  and  B)  but  not  for  non-system  technologies,  such  as  processes,  methods,  algo¬ 
rithms,  or  architectures.  Based  on  the  SEI’s  experience  in  transitioning  new  software  engi¬ 
neering  practices  (an  example  of  a  non-system  technology),  we  believe  that  the  TRLs  should 
be  extended  to  include  new  corollaries  for  these  kinds  of  technologies.  However,  the  Army 
STO  Manager  informed  us  that  the  majority  of  IA  technologies  they  are  currently  evaluating 
are  software.  Thus,  TRLs  for  software  (or  a  derivative)  should  be  sufficient  for  many  of  the 
technologies  they  evaluate. 

With  the  above  information,  what  is  our  answer  to  our  conceptual  feasibility  study:  Is  it  fea¬ 
sible  for  TRLs  (or  an  equivalent)  to  support  or  improve  the  STO  Manager’s  decision-making 
process  in  selecting  new  IA  technologies  for  investment? 


Yes,  it  is  feasible  for  TRLs  (or  an  equivalent )  to  support  or  add  value  to  the  decision-making 
process.  However,  it  is  only  one  of  several  critical  factors  in  the  decision-making  process, 
and,  as  currently  defined  for  system  technologies,  it  is  insufficient  for  use  with  non-system 
technologies  such  as  processes,  methods,  algorithms,  and  architectures. 

With  this  affirmative  result,  we  now  turn  to  our  findings  on  developing  a  tool  to  facilitate 
TRL  assessments. 
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2.3  Development  Feasibility 

This  section  details  our  findings  in  response  to  the  Development  Feasibility  question:  What 
resources  are  needed  to  develop  a  TRL  tool  or  adapt  an  existing  TRL  tool  for  STO  use? 

We  investigated  this  question  by  talking  with  users  of  TRLs  from  the  GAO,  AFRL,  and 
DARPA  to  discover  how  they  have  implemented  TRLs.  The  findings  are  interesting.  The 
range  of  implementation  approaches  is  broad,  ranging  from  a  formal  software  tool,  i.e.,  the 
“TRL  Calculator”  developed  at  AFRL,  to  more  informal  face-to-face  discussions  between  the 
stakeholders. 


AFRL  has  been  using  TRLs  for  about  three  years.7  The  GAO  has  also  been  involved  in  TRL 
assessments  since  the  release  of  their  Best  Practices  report  [GAO/NSIAD  99]  in  1999.  Two 
DARPA  Program  Managers  we  interviewed8  had  used  TRLs  on  more  than  seventy  IA  pro¬ 
jects.  The  major  findings  from  our  interviews  with  AFRL,  GAO,  and  DARPA  personnel9  who 
have  used  TRLs  are  as  follows: 

•  Some  of  the  interviewees  suggested  that  the  greatest  value  from  using  TRLs  comes  from 
the  discussions  between  the  stakeholders  that  go  into  negotiating  a  TRL  value. 

•  TRLs  provide  a  common  language  between  the  technology  developers,  program  office, 
and  engineers  who  will  adopt/use  the  technology. 

•  An  objective  observer  adds  to  the  TRL  accuracy  and  thus  to  its  utility. 

•  Currently,  there  is  no  standard  or  commonly  used  approach  for  implementing  TRLs. 

-  The  AFRL  personnel  we  interviewed  use  either  the  TRL  Calculator  tool  to  conduct 
their  assessments  or  hold  discussions  between  the  technology  stakeholders. 

-  The  GAO  personnel  we  interviewed  generally  gather  the  stakeholders  in  a  room 
where  they  jointly  work  through  the  TRL  descriptions,  and  jointly  arrive  at  a  TRL 
decision  (a  process  that  sometimes  takes  up  to  two  days). 

-  The  DARPA  PMs  we  interviewed  use  small  teams  of  three  to  four  personnel  includ¬ 
ing  the  PI.  Estimates  by  each  team  member  are  generally  done  independently  of  each 
other  and  differences  are  reconciled  to  arrive  at  a  consensus. 

-  CECOM  RDEC  personnel  gather  the  STO  management  team  to  collectively  assess 
maturity  based  on  task  progress  and  laboratory  and  field  test  results. 

•  William  Nolte  at  AFRL  has  developed  a  TRL  Calculator  for  both  hardware  and  software 
that  has  been  made  available  to  STO. 

The  last  item  deserves  special  attention,  since  the  STO  Manager  is  ultimately  seeking  a  tool 
of  some  form  to  facilitate  STO  TRL  assessment.  The  only  TRL  tool  found  in  this  investiga¬ 
tion  was  the  TRL  Calculator  (for  hardware  and  for  software)  developed  by  Mr.  Nolte  at 
AFRL.  Mr.  Nolte  released  an  alpha  version  of  the  TRL  Calculator  (it  is  a  Microsoft  Excel 


7  Conversation  with  Jim  Harris,  AFRL/WSPT. 

8  Conversations  with  Doug  Maughan,  DARPA  and  Jay  Lala,  DARPA. 

9  Conversations  with  Mathew  Lea,  GAO;  Jim  Harris,  AFRL/WSPT;  William  Nolte,  AFRL;  Doug 
Maughan,  DARPA;  Jay  Lala,  DARPA. 
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application)  in  January  2002.  Further  refinements  led  to  a  beta  release  (vl.O)  in  March  2002, 
and,  most  recently,  version  1.1,  released  in  August  2002.  According  to  Mr.  Nolte,  the  TRL 
Calculator  for  hardware  is  based  on  NASA’s  TRL  definitions,  which  AFRL  adopted  three 
years  ago.  Figure  1  shows  a  screen  capture  from  the  TRL  Calculator  (vl.O)  page  of  questions. 


Software  TRL  Calculator  vl.O 

WKER 


Technology  Readiness  Level  Achieved 


Level  of  Knowledge  -  Check  all  that  apply  or  use  sutler  tor  "/a  complete 


i  i ....  [Know what  software  needs  to  do  in  general  terms _ _ 

Have  some  concept  in  mind  that  may  be  realizable  in  software _ _ 

*  If"  (Have  an  idea  that  captures  the  basic  principles  of  a  possible  algorithm 

Initial  analysis  shows  what  major  functions  need  to  be  done  in  software _ 

is  gives  some  idea  of  what  software  architecture  will  look  like _ 

Analysis  provides  detailed  knowledge  of  specific  functions  software  needs  to  perform 

[Outline  of  software  algorithms  available _ _ _ _ _ 

I !~  {Able  to  estimate  software  program  size  in  lines  of  code  and/or  function  points 

Know  what  software  is  presently  available  that  does  similar  task  (Inventory  completed) 
w. ,,  .Know  limitations  of  presently  available  software  (Analysis  of  current  software  completed) 

r  Know  what  hardware  software  will  be  hosted  on _ _ 

'  IT.  [Know  what  output  devices  areavailable 

Inventory  of  external  interfaces  completed _ _ _ 1 

Analysis  of  internal  interface  requirements  completed _ _ 

Analysis  of  timing  constraints  completed _ _ _ 

Analysis  of  data  requirements  and  formats  completed  _ _ 

Analysis  of  database  structures  and  interfaces  completed _ 

External  interfaces  described  as  to  source,  format,  structure,  content,  and  method  of  supi 


Sllil 

■7  v  ,  "4  ■■■;  *  jSspi 

•  -Vj"4v 


IllflfeS 


Know  what  program  software  will  support _  . _ 

Customer  identified  _ _ _ 

Customer  expresses  interest  in  application _ 

Customer  participates  in  requirements  generation _ 

Customer  publishes  requirements  document _ _ _ 

Customer  representative  is  member  of  Integrated  Product  Team  (IPT) 

Customer  identifies  transition  windowfs)  of  opportunity _ 

Customer  commits  to  transition  through  ATP  commissioning  and/or  MQU 


-  v  ; 

- 

’ 


P | Customer  commits  to  transition  via  POM  process 


ieaet  Develop  $ 


lltli- 


Itse 

mmm 

siwb|| 


Software  architecture  defined  in  terms  of  major  functions  to  be  performed 

Preliminary  algorithm  development  completed _ 

Software  programming  language  selected _ _ 


mwsmm 


Figure  1:  A  Screen  Image  from  the  Software  TRL  Calculator  Vl.O 


The  TRL  Calculator  for  software  is  based  on  the  Army’s  software  TRL  definitions,  with  some 
modifications.  While  the  tool  has  not  undergone  formal  verification  and  validation,  it  is  being 
used  within  AFRL  and  has  demonstrated  success  (meaning  the  calculations  are  producing 
TRL  values  that  the  stakeholders  agree  with),  according  to  Mr.  Nolte.  Because  of  the  latter, 
Mr.  Nolte  estimates  the  tool  is  itself  reflecting  a  TRL  7.  The  software  TRL  calculator  is  based 
on  the  Army’s  TRL  definitions  for  software.  The  STO  Manager  has  expressed  concern  that 
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some  members  of  the  software  community  view  the  Army’s  TRL  definitions  for  software  as 
too  restrictive,  particularly  with  regard  to  the  verification  and  validation  requirements.  Based 
on  a  similar  perspective,  Mr.  Nolte  developed  the  Software  TRL  Calculator  with  modifica¬ 
tions  to  the  TRL  definitions  for  software. 

An  independent  evaluation  of  the  tool  and  its  relevance  to  the  STO  context  is  beyond  the 
scope  of  this  project.  However,  to  allow  STO  personnel  to  evaluate  the  tool  for  themselves, 
the  tool  was  given  to  the  STO  Manager  in  May  2002  by  the  authors  of  this  report,  with  Mr. 
Nolte’s  permission. 


Our  literature  survey  uncovered  one  variation  to  the  TRLs,  written  by  DARPA  PM  Douglass 
Gage.  PM  Gage  suggests  the  following  refinements  to  the  TRL  scale: 


TRL  3.5 

Characterize  target  functionality,  performance,  costs — use  as  input  to  a  decision 
to  pursue  serious  technology  development 

TRL  5.5 

Validation/development/refinement  evaluation,  decision  refinement/ 
integration — use  as  input  to  a  decision  to  integrate  the  technology  into  a  system 

TRL  8.5 

Production/deployment 

The  interesting  aspect  of  this  adaptation  of  TRLs  is  not  so  much  the  definitions,  but  that  this 
is  a  sign  that  others  are  picking  up  on  the  TRL  idea  and  adapting  it  to  their  needs.  A  commu¬ 
nity  of  practice  built  around  the  use  of  TRLs  would  be  an  interesting  and  useful  way  to  accel¬ 
erate  the  sharing  of  this  kind  of  information.  Communities  of  practice  are  groups  of  people 
who  share  a  concern,  a  set  of  problems,  or  a  passion  about  a  topic,  and  who  deepen  their 
knowledge  and  expertise  in  this  area  by  interacting  on  an  ongoing  basis  [Wenger  02]. 

Based  on  the  above  findings,  what  is  the  answer  to  our  Development  Feasibility  question, 

W  hat  resources  are  needed  to  develop  a  TRL  tool  or  adapt  an  existing  TRL  tool  for  STO  use? 


The  only  TRL  tool  found  in  this  investigation  is  the  TRL  Calculator  (one  for  hardware  and 
one  for  software)  developed  by  Mr.  Nolte  at  AFRL.  STO  is  currently  evaluating  this  tool.  If 
the  tool  proves  useful  to  STO,  then  extending  it  to  account  for  non-system  technologies  such 
as  processes ,  methods,  algorithms,  and  architectures  should  be  considered.  This  would  first 
require  establishing  equivalent  TRL  definitions  for  these  types  of  technologies,  and  then,  if 
desired,  incorporating  those  into  the  TRL  Calculator  tool.  Mr.  Nolte  has  expressed  interest  in 
working  with  STO  to  apply  his  technology  in  their  domain. 

A  less  sophisticated  “tool”  that  could  prove  useful  would  be  to  define  and  develop  a  system¬ 
atic,  repeatable  approach  (i.e.,  a  process)  for  determining  TRLs  based  on  the  STO.  We  found 
no  such  defined  process  in  our  investigations.  If  desired,  the  TRL  Calculator  could  enhance 
the  process  by  facilitating  the  consensus-building  steps  in  a  TRL  process.  For  example,  the 
tool  asks  detailed  questions  to  calculate  a  TRL.  Comparing  the  answers  between  process  par- 
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ticipants  can  highlight  differences  of  opinion  and  allow  the  TRL  process  lead  to  focus  on  ad¬ 
dressing  those  differences.  This  has  the  potential  to  make  the  TRL  process  more  efficient ,  con¬ 
sistent,  and  reliable. 


2.4  TRL  Tool  Implementation  Feasibility 

This  section  details  our  findings  in  response  to  the  Transition  Feasibility  question:  What  re¬ 
sources  are  needed  to  employ  such  a  tool  in  the  Army  STO  and  DARPA  S&T  community? 


We  investigated  several  approaches  to  getting  a  TRL  tool  into  use  in  ways  that  satisfy  the 
overall  STO  objective  of  gaining  visibility  into  the  TRLs  of  the  many  (approximately  50)  IA 
technologies  under  development  in  DARPA  each  year.  These  include: 

•  In  the  case  of  RFPs,  require  the  submitters  to  include  a  TRL  estimate  in  their  proposal. 
This  may  require  additional  guidance  in  the  RFP  on  how  to  determine  a  TRL.  And  again, 
to  address  non-system  technologies  such  as  processes,  methods,  architectures,  and  algo¬ 
rithms,  TRL  corollaries  for  those  types  of  technologies  should  be  developed.  In  some  cir¬ 
cles  (Army)  this  is  occurring  in  the  form  of  Technology  Readiness  Assessments  (TRAs), 
but  without  the  aid  of  a  tool. 

•  Establish  a  team,  or  work  with  an  independent  organization,  to  regularly  assess  the  TRLs 
of  DARPA  technologies.  This  report  has  already  shown  that  TRLs  are  a  subjective  meas¬ 
ure,  sometimes  involving  lengthy  discussions  and  negotiations  between  the  interested 
parties  in  order  to  come  to  consensus  on  a  TRL  value,  and  that  an  objective  observer 
helps  to  get  accurate  values.  One  of  the  challenges  that  CECOM  has  is  that  the  need  to 
evaluate  so  many  technologies  can  result  in  excessive  time  spent  in  discussions  with  the 
technology  developers  and  the  CECOM  engineers  to  arrive  at  a  TRL  consensus.  A  team 
with  the  primary  responsibility  of  evaluating  TRLs  can  provide  the  labor  to  do  this.  They 
would  also  have  the  responsibility  to  continually  refine  the  process  by  which  TRLs  are 
agreed  on. 

•  Have  STO  personnel  use  the  TRL  Calculator  at  the  DARPA  PI  meetings  and  conferences 
where  technologies  are  reviewed  and  answer  the  questions  in  the  tool  as  they  listen  to  the 
presentations,  view  the  demos,  or  read  the  papers.  This  turns  out  not  to  be  a  reasonable 
option  since,  as  previously  stated,  these  meetings  may  not  provide  sufficient  detail  for 
each  technology  to  be  able  to  answer  the  TRL  Calculator  questions. 

•  Work  with  DARPA  to  get  them  to  include  TRL  estimates  in  the  documents  and  other  ma¬ 
terials  they  provide  at  the  PI  meetings  and  conferences.  This  sounds  simpler  than  it  is. 
Our  interviews  with  two  DARPA  program  managers  led  us  to  the  conclusion  that,  at  least 
in  the  near  term,  only  a  DARPA  policy  mandate  will  result  in  regular  application  of  TRLs 
by  DARPA  PMs  and  Pis  and  regular  inclusion  of  that  information  in  DARPA  technology 
materials.  The  two  DARPA  managers  that  we  interviewed  used  TRLs  with  more  than  70 
of  their  technologies  and  found  them  useful,  but  said  they  would  not  likely  use  them 
again  because  of  the  level  of  effort  expended  in  their  already  tight  schedules.  Senior 
DARPA  officials  mandated  the  first  use  of  TRLs.  Thus,  organizations  like  STO  that 
would  like  to  have  TRL  data  provided  in  the  information  package  from  DARPA  will 
probably  not  see  that  in  the  near  future. 
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3  Recommendations 


We  can  summarize  the  findings  from  the  Conceptual  Feasibility  study,  the  Development  Fea¬ 
sibility  study,  and  the  Transition  Feasibility  study  as  follows: 

•  The  TRL  scale  provides  utility  as  one  of  several  critical  factors  in  the  ATD/STO  technol¬ 
ogy  selection  process.  It  is  best  to  view  TRLs  as  a  risk-reduction  measure  in  conjunction 
with  the  other  criteria. 

•  As  currently  defined  for  system  components,  the  TRLs  could  be  much  better  defined  to 
account  for  the  uniqueness  of  non-system  technologies,  such  as  processes,  methods,  algo¬ 
rithms,  or  architecture.  This  was  recognized  and  mentioned  by  several  of  the  interview¬ 
ees. 

•  The  only  TRL  tool  found  in  this  investigation  is  the  TRL  Calculator  (one  for  hardware 
and  one  for  software)  developed  by  Mr.  Nolte  at  AFRL.  A  less  sophisticated  “tool”  would 
be  a  systematic,  repeatable  process  for  determining  TRLs  (though  we  found  no  such  de¬ 
fined  process  in  our  investigations).  The  TRL  Calculator  could  be  a  support  tool  in  such  a 
process. 

•  It  is  unlikely  that  DARPA  personnel  will  include  TRL  information  in  their  technology 
documents  (briefings,  papers,  etc.)  in  the  near  future,  unless  it  is  mandated  from  senior 
DARPA  officials.  This  could  aid  in  engaging  early  maturity  technologies,  but  in  the  more 
exploratory  stages  of  research,  i.e.,  lower  TRLs,  the  estimate  may  be  somewhat  subjec¬ 
tive. 

•  While  the  TRL  Calculator  provides  a  repeatable  set  of  questions  for  determining  a  TRL, 
it  is  the  negotiation  of  the  answers  that  is  labor  intensive.  Thus,  a  good  consensus¬ 
building  and  conflict-resolution  process  is  also  needed. 

•  Much  of  the  value  of  TRLs  comes  from  the  discussions  between  the  stakeholders  that  go 
into  negotiating  the  TRL  value. 

Based  on  these  findings  we  offer  the  following  recommendations: 

•  Include  relevant  technology  stakeholders  in  TRL  negotiations. 

•  Develop  an  efficient  process  for  negotiating  TRLs  with  the  relevant  technology  stake¬ 
holders.  Use  the  TRL  Calculator  tool  to  support  the  process  (if  the  STO  evaluations  cur¬ 
rently  underway  report  favorably  on  the  tool  for  their  context). 

•  Extend  the  utility  of  TRLs  by  developing  corollaries  for  non-system  LA  technologies, 
such  as  processes,  methods,  architectures,  and  algorithms. 

•  Include  TRL  language  in  RFPs  and  provide  guidance  on  how  to  calculate  a  TRL. 
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Appendix  A  Technology  Readiness 

Levels 


In  their  work  on  best  business  practices  in  the  last  few  years,  the  GAO  studied  a  number  of 
commercial  firms  to  determine  key  factors  in  successful  product  development.  They  reported 
that  one  such  key  factor  is  maturing  a  new  technology  far  enough  to  get  it  into  the  right  size, 
weight,  and  configuration  needed  for  the  intended  product.  After  this  is  demonstrated,  the 
technology  is  said  to  be  at  an  acceptable  level  for  product  development.  According  to  the 
GAO  [GAO  01],  “organizations  that  use  best  practices  recognize  that  delaying  the  resolution 
of  technology  problems  until  product  development — analogous  to  the  engineering  and  manu¬ 
facturing  development  phase — can  result  in  at  least  a  ten-fold  cost  increase;  delaying  the 
resolution  until  after  the  start  of  production  could  increase  costs  by  a  hundred-fold.”  To  illus¬ 
trate  their  point,  the  same  report  cites  an  assessment  of  the  readiness  of  critical  technologies 
for  the  Joint  Strike  Fighter  program  and  makes  a  comparison  between  the  success  of  the  Joint 
Direct  Attack  Munition  and  the  Comanche  helicopter  programs. 

“For  example,  the  Joint  Direct  Attack  Munition  (JDAM)  used  modified 
variants  of  proven  components  for  guidance  and  global  positioning.  It  also 
used  mature,  existing  components  from  other  proven  manufacturing  proc¬ 
esses  for  its  own  system  for  controlling  tail  fin  movements.  The  munition 
was  touted  for  its  performance  in  Kosovo  and  was  purchased  for  less  than 
half  of  its  expected  unit  cost.  However,  the  Comanche  helicopter  program 
began  with  critical  technologies  such  as  the  engine,  rotor,  and  integrated 
avionics  at  TRL  levels  of  5  or  below.  That  program  has  seen  101  percent 
cost  growth  and  120-percent  schedule  slippage  as  a  result  of  these  low 
maturity  levels  and  other  factors.” 

To  improve  the  ability  of  programs  to  select  mature  technologies  for  inclusion  in  their  pro¬ 
grams,  the  GAO  recommended  the  use  of  Technology  Readiness  Levels  (TRL).  TRLs  were 
pioneered  by  the  National  Aeronautics  and  Space  Administration  and  adopted  by  the  Air 
Force  Research  Laboratory  (AFRL),  which  promotes  them  as  a  means  of  evaluating  the 
readiness  of  technologies  to  be  incorporated  into  a  weapon  or  other  type  of  system.  TRLs  are 
being  promoted  as  a  gap  assessment  between  a  technology’s  current  maturity  and  the  matur¬ 
ity  needed  for  successful  inclusion.  The  AFRL  judges  a  technology  to  be  low  risk  for  the  en¬ 
gineering  and  manufacturing  development  stage  when  (a)  a  prototype  of  that  technology  has 
been  developed  that  includes  all  of  its  critical  components  in  approximately  the  same  size  and 
weight,  and  ( b )  that  prototype  has  been  demonstrated  to  work  in  an  environment  similar  to 
that  of  the  planned  operational  system  [GAO  01]. 
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TRLs  follow  a  scale  from  1  (lowest  level  of  readiness)  to  9  (mature  development).  For  exam¬ 
ple,  a  technology  assessed  at  TRL  1  is  by  definition  at  the  lowest  level  of  technology  readi¬ 
ness,  “where  scientific  research  begins  to  be  translated  into  applied  research  and  develop¬ 
ment”  [GAO/NSIAD  99].  By  the  time  the  technology  has  reached  a  TRL  9,  the  technology 
has  progressed  through  formulation  of  an  initial  concept  for  application,  proof  of  concept, 
demonstration  in  a  laboratory  environment  and  realistic  environment,  and  integration  into  a 
system,  and  has  been  “flight  qualified”  and  then  “flight  proven.”  This  last  state  of  develop¬ 
ment,  where  the  technology  is  operating  under  mission  conditions,  is  TRL  9.  The  AFRL  con¬ 
siders  TRL  7  to  be  an  acceptable  risk  for  starting  the  engineering  and  manufacturing  devel¬ 
opment  phase. 

In  a  July  15  2001  memorandum,  the  Deputy  Under  Secretary  of  Defense  (Science  and  Tech¬ 
nology)  officially  endorsed  the  use  of  TRLs  in  new  major  programs.  New  DoD  regulations 
require  that  the  military  services’  science  and  technology  executives  conduct  a  technology 
readiness  level  assessment  for  critical  technologies  identified  in  major  weapon  systems  pro¬ 
grams  prior  to  the  start  of  engineering  and  manufacturing  development  and  production.  The 
memorandum  notes  that  technology  readiness  levels  are  the  preferred  approach  for  all  new 
major  programs  unless  the  Deputy  Under  Secretary  approves  an  equivalent  assessment 
method. 

Table  1  is  an  excerpt  from  the  DoD  5000.2-R  document  [DoD  02],  which  specifies  TRLs 
from  a  systems  approach.  TRLs  thus  are  intended  to  be  appropriate  for  both  hardware  and 
software.  The  document  also  states  “DoD  Components  may  provide  additional  clarifications 
for  software.” 


Table  1:  TRL  Descriptions 


Technology  Readiness  Level 

Description 

1.  Basic  principles  observed 
and  reported 

Lowest  level  of  technology  readiness.  Scientific  research 
begins  to  be  translated  into  applied  research  and  devel¬ 
opment.  Examples  might  include  paper  studies  of  a 
technology’s  basic  properties. 

2.  Technology  concept  and/or 
application  formulated 

Invention  begins.  Once  basic  principles  are  observed, 
practical  applications  can  be  invented.  Applications  are 
speculative  and  there  may  be  no  proof  or  detailed  analy¬ 
sis  to  support  the  assumptions.  Examples  are  limited  to 
analytic  studies. 

3.  Analytical  and  experimental 
critical  function  and/or  char¬ 
acteristic  proof  of  concept 

Active  research  and  development  is  initiated.  This  in¬ 
cludes  analytical  studies  and  laboratory  studies  to  physi¬ 
cally  validate  analytical  predictions  of  separate  elements 
of  the  technology.  Examples  include  components  that  are 
not  yet  integrated  or  representative. 

4.  Component  and/or  bread¬ 
board  validation  in  labora- 

Basic  technological  components  are  integrated  to  estab¬ 
lish  that  they  will  work  together.  This  is  relatively  “low 

16 


CMU/SEI-2002-SR-027 


Technology  Readiness  Level 

Description 

tory  environment 

fidelity”  compared  to  the  eventual  system.  Examples 
include  integration  of  “ad  hoc”  hardware  in  the  labora¬ 
tory. 

5.  Component  and/or  bread¬ 
board  validation  in  relevant 
environment 

Fidelity  of  breadboard  technology  increases  signifi¬ 
cantly.  The  basic  technological  components  are  inte¬ 
grated  with  reasonably  realistic  supporting  elements  so  it 
can  be  tested  in  a  simulated  environment.  Examples  in¬ 
clude  “high-fidelity”  laboratory  integration  of  compo¬ 
nents. 

6.  System/subsystem  model  or 
prototype  demonstration  in  a 
relevant  environment 

Representative  model  or  prototype  system,  which  is  well 
beyond  that  of  TRL  5,  is  tested  in  a  relevant  environ¬ 
ment.  Represents  a  major  step  up  in  a  technology’s  dem¬ 
onstrated  readiness.  Examples  include  testing  a  proto¬ 
type  in  a  high-fidelity  laboratory  environment  or  in  a 
simulated  operational  environment. 

7.  System  prototype  demonstra¬ 
tion  in  an  operational  envi¬ 
ronment 

Prototype  near,  or  at,  planned  operational  system.  Repre¬ 
sents  a  major  step  up  from  TRL  6,  requiring  demonstra¬ 
tion  of  an  actual  system  prototype  in  an  operational  envi¬ 
ronment  such  as  an  aircraft,  vehicle,  or  space.  Examples 
include  testing  the  prototype  in  a  test  bed  aircraft. 

8.  Actual  system  completed  and 
qualified  through  test  and 
demonstration 

Technology  has  been  proven  to  work  in  its  final  form 
and  under  expected  conditions.  In  almost  all  cases,  this 
TRL  represents  the  end  of  true  system  development.  Ex¬ 
amples  include  developmental  test  and  evaluation  of  the 
system  in  its  intended  weapon  system  to  determine  if  it 
meets  design  specifications. 

9.  Actual  system  proven 
through  successful  mission 
operations 

Actual  application  of  the  technology  in  its  final  form  and 
under  mission  conditions,  such  as  those  encountered  in 
operational  test  and  evaluation.  Examples  include  using 
the  system  under  operational  mission  conditions. 

Definitions 

Breadboard :  Integrated  components  that  provide  a  representation  of  a  system/subsystem  and 
that  can  be  used  to  determine  concept  feasibility  and  to  develop  technical  data.  Typically  con¬ 
figured  for  laboratory  use  to  demonstrate  the  technical  principles  of  immediate  interest.  May 
resemble  final  system/subsystem  in  function  only. 

High  fidelity:  Addresses  form,  fit,  and  function.  High-fidelity  laboratory  environment  would 
involve  testing  with  equipment  that  can  simulate  and  validate  all  system  specifications  within 
a  laboratory  setting. 
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Low  fidelity:  A  representative  of  the  component  or  system  that  has  limited  ability  to  provide 
anything  but  first  order  information  about  the  end  product.  Low-fidelity  assessments  are  used 
to  provide  trend  analysis. 

Model:  A  functional  form  of  a  system,  generally  reduced  in  scale,  near  or  at  operational  speci¬ 
fication.  Models  will  be  sufficiently  hardened  to  allow  demonstration  of  the  technical  and 
operational  capabilities  required  of  the  final  system. 

Operational  environment:  Environment  that  addresses  all  of  the  operational  requirements  and 
specifications  required  of  the  final  system,  including  platform/packaging. 

Prototype:  A  physical  or  virtual  model  used  to  evaluate  the  technical  or  manufacturing  feasi¬ 
bility  or  military  utility  of  a  particular  technology  or  process,  concept,  end  item,  or  system. 

Relevant  environment:  Testing  environment  that  simulates  the  key  aspects  of  the  operational 
environment. 

Simulated  operational  environment:  Either  (a)  a  real  environment  that  can  simulate  all  of  the 
operational  requirements  and  specifications  required  of  the  final  system,  or  ( b )  a  simulated 
environment  that  allows  for  testing  of  a  virtual  prototype.  Used  in  either  case  to  determine 
whether  a  developmental  system  meets  the  operational  requirements  and  specifications  of  the 
final  system. 
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Appendix  B  Army  Draft  of  TRLs  for 

Software 


Table  2  contains  an  excerpt  from  a  document  provided  by  Mathew  Lea  of  the  GAO.  Mr. 
Lea’s  information  came  from  CECOM  Research  Development  and  Engineering  Center.  For 
each  TRL,  descriptions  are  given  for  hardware/subsystems  (HW/S),  and  software  (SW). 

Table  2:  TRL  Descriptions  for  Hardware/Subsystems  and  Software 


Technology  Readiness  Level 

Description 

1.  Basic  principles  observed  and 
reported 

HW/S:  Lowest  level  of  technology  readiness.  Scientific 
research  begins  to  be  translated  into  applied  research  and 
development.  Examples  might  include  paper  studies  of  a 
technology’s  basic  properties. 

SW:  Lowest  level  of  software  readiness.  Basic  research 
begins  to  be  translated  into  applied  research  and  devel¬ 
opment.  Examples  might  include  a  concept  that  can  be 
implemented  in  software  or  analytic  studies  of  an  algo¬ 
rithm’s  basic  properties. 

2.  Technology  concept  and/or 
application  formulated 

HW/S/SW:  Invention  begins.  Once  basic  principles  are 
observed,  practical  applications  can  be  invented.  Applica¬ 
tions  are  speculative  and  there  may  be  no  proof  or  de¬ 
tailed  analysis  to  support  the  assumptions.  Examples  are 
limited  to  analytic  studies. 

3.  Analytical  and  experimental 
critical  function  and/or  char- 
acteristic  proof  of  concept 

HW/S:  Active  research  and  development  is  initiated.  This 
includes  analytical  studies  and  laboratory  studies  to 
physically  validate  analytical  predictions  of  separate 
elements  of  the  technology.  Examples  include  compo¬ 
nents  that  are  not  yet  integrated  or  representative. 

SW:  Active  research  and  development  is  initiated.  This 
includes  analytical  studies  to  produce  code  that  validates 
analytical  predictions  of  separate  software  elements  of 
the  technology.  Examples  include  software  components 
that  are  not  yet  integrated  or  representative  but  satisfy  an 
operational  need.  Algorithms  run  on  a  surrogate  proces¬ 
sor  in  a  laboratory  environment. 

4.  Component  and/or  bread¬ 
board  validation  in  laboratory 
environment 

HW/S:  Basic  technological  components  are  integrated  to 
establish  that  they  will  work  together.  This  is  relatively 
“low  fidelity”  compared  to  the  eventual  system.  Exam¬ 
ples  include  integration  of  ad  hoc  hardware  in  the  labora- 
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Technology  Readiness  Level 


5.  Component  and/or  bread¬ 
board  validation  in  relevant 
environment 


6.  System/subsystem  model  or 
prototype  demonstration  in  a 
relevant  environment 


7.  System  prototype  demonstra¬ 
tion  in  an  operational  envi¬ 
ronment 


Description 


tory. 

SW:  Basic  software  components  are  integrated  to  estab¬ 
lish  that  they  will  work  together.  They  are  relatively 
primitive  with  regard  to  efficiency  and  reliability  com¬ 
pared  to  the  eventual  system.  System  software  architec¬ 
ture  development  initiated  to  include  interoperability, 
reliability,  maintainability,  extensibility,  scalability,  and 
security  issues.  Software  integrated  with  simulated  cur¬ 
rent/legacy  elements  as  appropriate. 


HW/S:  Fidelity  of  breadboard  technology  increases  sig¬ 
nificantly.  The  basic  technological  components  are  inte¬ 
grated  with  reasonably  realistic  supporting  elements  so  it 
can  be  tested  in  a  simulated  environment.  Examples  in¬ 
clude  “high  fidelity”  laboratory  integration  of  compo¬ 
nents. 

SW:  Reliability  of  software  ensemble  increases  signifi¬ 
cantly.  The  basic  software  components  are  integrated 
with  reasonably  realistic  supporting  elements  so  that  it 
can  be  tested  in  a  simulated  environment.  Examples  in¬ 
clude  “high  fidelity”  laboratory  integration  of  software 
components. 

System  software  architecture  established.  Algorithms  run 
on  a  processor(s)  with  characteristics  expected  in  the 
operational  environment.  Software  releases  are  “Alpha” 
versions  and  configuration  control  is  initiated.  Verifica¬ 
tion,  Validation,  and  Accreditation  (VV&A)  initiated. 


HW/S:  Representative  model  or  prototype  system,  which 
is  well  beyond  that  of  TRL  5,  is  tested  in  a  relevant  envi¬ 
ronment.  Represents  a  major  step  up  in  a  technology’s 
demonstrated  readiness.  Examples  include  testing  a  pro¬ 
totype  in  a  high-fidelity  laboratory  environment  or  in  a 
simulated  operational  environment. 

SW:  Representative  model  or  prototype  system,  which  is 
well  beyond  that  of  TRL  5,  is  tested  in  a  relevant  envi¬ 
ronment.  Represents  a  major  step  up  in  software- 
demonstrated  readiness.  Examples  include  testing  a  pro¬ 
totype  in  a  live/virtual  experiment  or  in  a  simulated  op¬ 
erational  environment.  Algorithms  run  on  processor  of 
the  operational  environment  are  integrated  with  actual 
external  entities.  Software  releases  are  “Beta”  versions 
and  configuration  controlled.  Software  support  structure 
is  in  development.  VV&A  is  in  process. 


HW/S:  Prototype  near,  or  at,  planned  operational  system. 
Represents  a  major  step  up  from  TRL  6,  requiring  dem¬ 
onstration  of  an  actual  system  prototype  in  an  operational 
environment  such  as  an  aircraft,  vehicle,  or  space.  Exam- 
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Technology  Readiness  Level 


8.  Actual  system  completed  and 
qualified  through  test  and 
demonstration 


9.  Actual  system  proven 
through  successful  mission 
operations 


Description 


pies  include  testing  the  prototype  in  a  test  bed  aircraft. 

SW:  Represents  a  major  step  up  from  TRL  6,  requiring 
the  demonstration  of  an  actual  system  prototype  in  an 
operational  environment,  such  as  in  a  command  post  or 
air/ground  vehicle.  Algorithms  run  on  processor  of  the 
operational  environment  are  integrated  with  actual  exter¬ 
nal  entities.  Software  support  structure  is  in  place.  Soft¬ 
ware  releases  are  in  distinct  versions.  Frequency  and  se¬ 
verity  of  software  deficiency  reports  do  not  significantly 
degrade  functionality  or  performance.  VV&A  completed. 


HW/S:  Technology  has  been  proven  to  work  in  its  final 
form  and  under  expected  conditions.  In  almost  all  cases, 
this  TRL  represents  the  end  of  true  system  development. 
Examples  include  developmental  test  and  evaluation  of 
the  system  in  its  intended  weapon  system  to  determine  if 
it  meets  design  specifications. 

SW:  Software  has  been  demonstrated  to  work  in  its  final 
form  and  under  expected  conditions.  In  most  cases,  this 
TRL  represents  the  end  of  system  development.  Exam¬ 
ples  include  test  and  evaluation  of  the  software  in  its  in¬ 
tended  system  to  determine  if  it  meets  design  specifica¬ 
tions.  Software  releases  are  production  versions  and 
configuration  controlled,  in  a  secure  environment.  Soft¬ 
ware  deficiencies  are  rapidly  resolved  through  support 
infrastructure. 


HW/S:  Actual  application  of  the  technology  in  its  final 
form  and  under  mission  conditions,  such  as  those  en¬ 
countered  in  operational  test  and  evaluation.  Examples 
include  using  the  system  under  operational  mission  con¬ 
ditions. 

SW:  Actual  application  of  the  software  in  its  final  form 
and  under  mission  conditions,  such  as  those  encountered 
in  operational  test  and  evaluation.  In  almost  all  cases,  this 
is  the  end  of  the  last  “bug  fixing”  aspects  of  the  system 
development.  Examples  include  using  the  system  under 
operational  mission  conditions.  Software  releases  are 
production  versions  and  configuration  controlled.  Fre¬ 
quency  and  severity  of  software  deficiencies  are  at  a 
minimum. 
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Appendix  C  Project  Plan 


The  work  statement  for  this  project  includes  three  phases  of  work: 

Phase  1:  Feasibility  Study 

The  SEI,  in  collaboration  with  the  Manager  of  STO,  will  determine  the  feasibility  of  develop¬ 
ing  an  IA  TRL  assessment  method  (or  equivalent)  for  technologies  at  various  TRLs.  This  will 
assist  the  Manager  of  the  STO  in  identifying  technologies  in  basic  research  and  applied  re¬ 
search  categories  that  would  benefit  the  Army.  If  determined  to  be  feasible,  the  SEI  will  col¬ 
laborate  with  the  STO  in  Phases  2  and  3  to  develop  and  transition  the  method  or  tool  into  use. 

Phase  2:  Assessment  Method  Development 

The  particular  assessment  method  to  be  developed  will  be  based  on  the  findings  from  the  fea¬ 
sibility  study.  However,  it  is  important  that  the  development  be  a  collaborative  effort  between 
SEI,  STO,  and  selected  DARPA  personnel  who  are  involved  in  the  development  of  a  technol¬ 
ogy  that  may  eventually  be  transitioned  into  the  Army. 

Phase  3:  Implementation  Support 

We  expect  that  the  implementation  of  a  new  technology  readiness  assessment  method  will 
include  a  focused  effort  on  the  planning  and  managing  of  its  transition  into  use.  The  SEI  will 
collaborate  with  STO  to  establish  and  implement  a  transition  plan  and  management  approach 
that  will  address  both  common  technology  adoption  issues  and  those  specific  to  the  new  as¬ 
sessment  method  and  the  organizations  that  will  be  adopting  it.  The  level  of  effort  for  this 
task  for  the  SEI  and  the  adopting  organizations  will  be  determined  at  the  conclusion  of  the 
feasibility  study. 

The  plan  for  Phase  1  Feasibility  Study  consisted  of  three  major  tasks:  (1)  assess  the  state  of 
the  TRL  practice  through  interviews  with  TRL  users  and  a  literature  survey,  (2)  identify  the 
needs  of  the  STO  with  regard  to  TRL  use,  and  (3)  synthesize  the  results  into  a  report  of  find¬ 
ings  and  recommendations.  The  status  of  each  of  these  activities  is  provided  in  Table  3. 
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Table  3:  Feasibility  Study  Activities 

Activity 


Progress  Status 


1.  Assess  the  state  of  the  practice  Activity  1  completed 

1.1  Literature  Survey 

•  DoD  Web  sites  and  literature 

•  Army  Web  sites  and  literature 

•  Navy  Web  sites  and  literature 

•  Air  Force  Web  sites  and  literature 

•  GAO  Web  sites  and  literature 

•  Articles  from  the  field  of  new  product  development 

1.2  Interviews  with  TRL  users 

•  GAO  /  Matt  Lea 

•  AFRL  /  Jim  Harris 

•  ARFL  /  Bill  Nolte 

•  DARPA  /  Jay  Lala 

•  DARPA  /  Doug  Maughan 

2.  Assess  the  needs  of  the  STO  Activity  2  completed 

•  Interview  current  ATD  developers 

•  Compile  interview  notes 

•  Interviewees  review  and  comment  on  notes 

3.  Synthesize  Findings  into  Report  Activity  3  completed 

3.1  Preliminary  Report 

•  Draft  preliminary  findings  and  recommendations 
report 

•  Bob,  Peter,  SEI  team  discuss  preliminary  report  and 
next  steps 

3.2  Final  Report 

•  Formulate  draft  of  final  report 

•  Bob,  Peter,  SEI  team  discuss  final  report  draft 

•  Refine  Final  Report  draft 

•  Deliver  Final  Report  to  Bob  and  Peter 
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